Skip to content

Replace legacy pipeline with token-efficient semantic index - #1

Merged
fajarnugraha37 merged 134 commits into
mainfrom
agent/token-efficient-semantic-index
Jul 16, 2026
Merged

Replace legacy pipeline with token-efficient semantic index#1
fajarnugraha37 merged 134 commits into
mainfrom
agent/token-efficient-semantic-index

Conversation

@fajarnugraha37

@fajarnugraha37 fajarnugraha37 commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Summary

Replaces the provider-heavy v1 workflow with DocGen 2.0, a breaking semantic-index architecture for repositories of any language, framework, library, runtime, infrastructure stack, or architecture.

Providers receive bounded, content-addressed context packs. Deterministic code owns source discovery, indexing, checkpoints, validation, quality gates, budgets, telemetry, and publishing.

Technology-neutral architecture

DocGen does not assume JAX-RS, Jersey, MyBatis, SQL, Kafka, Java, or any other particular stack. It supports applications, libraries, CLIs, jobs, plugins, infrastructure, data pipelines, embedded systems, monoliths, services, and mixed repositories.

The semantic index always records generic file artifacts and source chunks, then adds cross-language hints when available:

  • symbols, functions, modules/imports, interfaces, contracts, configuration keys, and URLs
  • npm, Maven, Gradle, .NET, Go, Cargo, Python, Ruby, Composer, and Mix manifests
  • Docker and Terraform artifacts
  • optional interface, data, automation, channel, schedule, and security-boundary signals

Unknown technologies remain supported through bounded source chunks rather than being rejected.

Phase 2A — runtime stabilization

  • one index phase per full run
  • explicit phase, model, executable, effective max-turns, timeout, context-size, and log-path output
  • hard minimum of 30 provider turns, including auto-healing legacy project configuration and environment overrides
  • running/completed/failed page checkpoints
  • crash-safe partial-batch recovery
  • provider non-zero recovery only for complete artifacts produced by the current invocation
  • hash plus nanosecond-mtime freshness proof to reject stale pre-existing artifacts
  • retries limited to unresolved pages instead of repeating completed work
  • resumable model, plan, generation, audit, and publish stages
  • timeout and process-tree termination on Windows and Unix
  • content-addressed reuse for unchanged reruns

Phase 2B — correctness validation

A new deterministic quality engine validates:

  • canonical inventory membership and live-source hashes
  • evidence paths and line ranges
  • FACT evidence requirements
  • semantic-model JSON, qualified identities, classifications, confidence, and evidence
  • Markdown frontmatter, H1, required sections, Mermaid policy, and local links
  • page, traceability, input, context, model, and artifact hashes
  • context-bound evidence and model references
  • missing, duplicate, conflicting, stale, orphaned, or substantially duplicated artifacts
  • model-reference coverage and configurable failure/warning policies
  • freshness of the audit before publishing

Publishing now fails when source, model, page, traceability, link, evidence, or audit inputs are stale.

Token-efficiency architecture

  • deterministic .gitignore / .docgenignore / binary-aware inventory
  • incremental SQLite/FTS5 index
  • bounded retrieval with explicit omission reporting
  • hard provider-call and token budgets
  • JSONL provider telemetry and per-stage usage summaries
  • deterministic rendering for low-risk references
  • item-level page input hashes
  • direct bounded prompts with no parent-agent delegation
  • selective, hash-cached semantic-risk audit

Validation

The regression suite now contains 20 tests covering:

  • real Git repositories and installed launcher execution
  • incremental indexing and bounded context
  • cross-language indexing without a required stack
  • one-index full pipeline and zero-call unchanged resume
  • minimum 30-turn enforcement
  • provider exit after valid output
  • partial generation checkpoints and selective retries
  • rejection of stale recovery artifacts
  • invalid evidence paths and line ranges
  • source changes after indexing
  • unknown model references
  • stale-source publish rejection
  • Windows command shim, heartbeat, timeout, and CRLF prompt handling
  • migration preservation

The complete suite and installer dry run pass on:

  • Ubuntu, Node.js 22
  • Ubuntu, Node.js 24
  • Windows Server 2025, Node.js 24

Breaking migration

Version remains 2.0.0; Node.js 22.5+ is required for node:sqlite.

Existing repositories run:

docgen migrate
docgen doctor
docgen all

Migration preserves generated docs, .docgenignore, project identity, selected provider settings, and ignore/binary limits. Legacy workflow state is archived rather than interpreted as a current checkpoint.

Status

Phase 2A and Phase 2B implementation is complete. The PR remains draft and unmerged pending validation against the real target repository and the Phase 2C cold/warm/incremental token benchmark.

fajarnugraha37 and others added 28 commits July 17, 2026 01:06
@fajarnugraha37
fajarnugraha37 marked this pull request as ready for review July 16, 2026 22:13
@fajarnugraha37
fajarnugraha37 merged commit e9e247c into main Jul 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant